Image capturing device and hybrid image processing method thereof

ABSTRACT

An image capturing device and a hybrid image processing method thereof are provided. The method is adapted to an image capturing device having a first lens and a second lens and includes the following steps. First, a scene is captured by the first lens and the second lens to respectively generate a mono image and a color image of the scene. Next, one of a mono image-based mode, a color image-based mode, and a color image-only mode is dynamically selected, and an output image is generated accordingly, wherein the mono image-based mode is to produce the output image by adding color image data onto the mono image, the color image-based mode is to produce the output image by adding mono image data onto the color image, and the color image-only mode is to produce the output image by only using the color image.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of U.S. provisionalapplication Ser. No. 62/161,271, filed on May 14, 2015 and Taiwanapplication serial no. 104126020, filed on Aug. 11, 2015. The entiretyof each of the above-mentioned patent applications is herebyincorporated by reference herein and made a part of this specification.

TECHNICAL FIELD

The disclosure relates to a hybrid image processing technique.

BACKGROUND

With development in technology, various smart mobile electronic devices,such as tablet computers, personal digital assistants and smart phones,have become indispensable tools for people nowadays. Camera lensesequipped in high-end smart mobile electronic devices provide same orbetter specifications than those of traditional consumer cameras, andsome even provide three-dimensional image capturing features ornear-equivalent pixel qualities to those of digital single lens reflexcameras. However, as opposed to slim design of mobile electronicdevices, the sizes of lens modules and their sensing elements wouldinevitably increase for attaining high-quality and high-resolutionimages.

SUMMARY OF THE INVENTION

Accordingly, the disclosure is directed to an image capturing device anda hybrid image processing method thereof, which not only provide outputimages with high quality but also accentuate the slimness of the imagecapturing device.

A hybrid image processing method of an image capturing device, adaptedto an image capturing device having a first lens and a second lens, isprovided in the disclosure. The method includes the following steps.First, a scene is captured by using the first lens and the second lensso as to respectively generate a mono image and a color image of thescene. Next, one of a mono image-based mode, a color image-based mode,and a color image-only mode is selected dynamically according to acontent of the scene so as to generate an output image. The aforesaidmono image-based mode is to produce the output image by adding colorimage data of the color image onto the mono image. The aforesaid colorimage-based mode is to produce the output image by adding mono imagedata of the mono image onto the color image. The aforesaid colorimage-only mode is to produce the output image by only using the colorimage.

According to an embodiment of the disclosure, the step of selecting oneof the mono image-based mode, the color image-based mode, and the colormage-only mode dynamically according to the content of the sceneincludes obtaining depth range and depth variation of the sceneaccording to the mono image and the color image and selecting one of themono image-based mode, the color image-based mode, and the colorimage-only mode dynamically according to at east one of the depth rangeand the depth variation.

According to an embodiment of the disclosure, the step of selecting oneof the mono image-based mode, the color image-based mode, and the colorimage-only mode dynamically according to at least one of the depth rangeand the depth variation includes as follows. Whether the depth range isgreater than a distance threshold or the depth variation is less than afirst depth variation threshold is determined. When the depth range isgreater than the distance threshold or the depth variation is less thanthe first depth variation threshold, the mono image-based mode isselected. When the depth range is not greater than the distancethreshold and the depth variation is not less than the first depthvariation threshold, whether the depth variation is less than a seconddepth variation threshold is determined, where the second depthvariation threshold is greater than the first depth variation threshold.When the depth variation is less than the second depth variationthreshold, the color image-based mode is selected. When the depthvariation is not less than the second depth variation threshold, thecolor image-only mode is selected.

According to an embodiment of the disclosure, the step of adding thecolor image data of the color image onto the mono image includessearching for a plurality of related color pixels from the color image,where the related color pixels are associated with luma pixels in themono image, and combining related color data of the related color pixelsinto luma data of the luma pixels.

According to an embodiment of the disclosure, the step of adding themono image data onto the color image includes searching for a pluralityof related luma pixels from the mono image, where the related lumapixels are associated with color pixels in the color image, andcombining related luma data of the related luma pixels into color dataof the color pixels.

An image capturing device is also provided in the disclosure. The imagecapturing device includes a first lens, a second lens, a storage unit,and one or more processing unit. The storage unit is coupled to thefirst lens and the second lens and configured to store images capturedby the first lens and the second lens. The processing unit is coupled tothe first lens, the second lens, and the storage unit and includesmultiple modules, where the modules include an image capturing module, amode selecting module, and an image processing module. The imagecapturing module captures a scene by using the first lens and the secondlens so as to respectively generate a mono image and a color image ofthe scene. The mode selecting module selects one of a mono image-basedmode, a color image-based mode, and a color image-only mode dynamicallyaccording to a content of the scene. The image processing modulegenerates an output image according to a mode selected by the modeselecting module, where the mono image-based mode is to produce theoutput image by adding color image data of the color image onto the monoimage, the color image-based mode is to produce the output image byadding mono image data of the mono image onto the color image, and thecolor image-only mode is to produce the output image by only using thecolor image.

According to an embodiment of the disclosure, the mode selecting moduleobtains depth range and depth variation of the scene according to themono image and the color image and selects one of the mono image-basedmode, the color image-based mode, and the color image-only modedynamically according to at least one of the depth range and the depthvariation.

According to an embodiment of the disclosure, the mode selecting moduledetermines whether the depth range is greater than a distance thresholdor the depth variation is less than a first depth variation threshold.When the depth range is greater than a distance threshold or the depthvariation is less than a first depth variation threshold, the modeselecting module selects the mono image-based mode. When the depth rangeis not greater than the distance threshold and the depth variation isnot less than the first depth variation threshold, the mode selectingmodule determines whether the depth variation is less than a seconddepth variation threshold, wherein the second depth variation thresholdis greater than the first depth variation threshold. When the depthvariation is less than a second depth variation threshold, the modeselecting module selects the color image-based mode. When the depthvariation is not less than the second depth variation threshold, themode selecting module selects the color image-only mode.

According to an embodiment of the disclosure, when the mode selectingmodule selects the mono image-based mode, the image processing modulesearches for a plurality of related color pixels from the color imageand combines related color data of the related color pixels into lumadata of the luma pixels, where the related color pixels are associatedwith luma pixels in the mono image.

According to an embodiment of the disclosure, when the mode selectingmodule selects the color image-based mode, the image processing modulesearches for a plurality of related luma pixels from the mono image andcombines related luma data of the related luma pixels into color data ofthe color pixels, where the related luma pixels are associated withcolor pixels in the color image.

In summary, in the proposed image capturing device and the hybrid imageprocessing method thereof, dual lenses are used for capturing a monoimage and a color image of a scene. An output image is generated bydynamically switching between the mono image-based mode, the colorimage-based mode, and the color image-only mode. Based on the advantagesof a mono image and a color image in different scenes, the disclosurewould adaptively switch to an image processing mode suitable for thecurrent scene, which not only provide output images with high qualitybut also accentuate the slimness of the image capturing device.

In order to make the aforementioned features and advantages of thepresent disclosure comprehensible, preferred embodiments accompaniedwith figures are described in detail below. It is to be understood thatboth the foregoing general description and the following detaileddescription are exemplary, and are intended to provide furtherexplanation of the disclosure as claimed.

It should be understood, however, that this summary may not contain allof the aspect and embodiments of the present disclosure and is thereforenot meant to be limiting or restrictive in any manner. Also the presentdisclosure would include improvements and modifications, which areobvious to one skilled in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the invention, and are incorporated in and constitute apart of this specification. The drawings illustrate embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention.

FIG. 1 illustrates a block diagram of an image capturing deviceaccording to an embodiment of the disclosure.

FIG. 2 illustrates a flowchart of a hybrid image processing method of animage capturing device according to an embodiment of the disclosure.

FIG. 3 illustrates a flowchart of a hybrid image processing method of animage capturing device according to an embodiment of the disclosure.

FIG. 4 illustrates a functional flowchart of a hybrid image processingmethod of an image capturing device according to an embodiment of thedisclosure.

To make the above features and advantages of the application morecomprehensible, several embodiments accompanied with drawings aredescribed in detail as follows.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the present embodiments of theinvention, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers are used in thedrawings and the description to refer to the same or like parts. Inaddition, the specifications and the like shown in the drawing figuresare intended to be illustrative, and not restrictive. Therefore,specific structural and functional detail disclosed herein are not to beinterpreted as limiting, but merely as a representative basis forteaching one skilled in the art to variously employ the invention.

FIG. 1 illustrates a block diagram of an image capturing deviceaccording to an embodiment of the disclosure. It should, however, benoted that this is merely an illustrative example and the invention isnot limited in this regard. All components of the image capturing deviceand their configurations are first introduced in FIG. 1. The detailedfunctionalities of the components are disclosed along with FIG. 2.

Referring to FIG. 1, an image capturing device 100 includes a first lens10 a, a second lens 10 b, a storage unit 15, and one or more processingunit 20. In the present embodiment, the image capturing device 100 maybe a digital camera, a digital camcorder, a digital single lens reflexcamera or other devices provided with an image capturing feature such asa smart phone, a tablet computer, a personal digital assistant, and soforth. The disclosure is not limited herein.

The first lens 10 a and the second lens 10 b include optical sensingelements for sensing light intensity entering the first lens 10 a andthe second lens 10 b to thereby generate images. The optical sensingelements are, for example, charge-coupled-device (CCD) elements,complementary metal-oxide semiconductor (CMOS) elements, and yet thedisclosure is not limited thereto. In the present embodiment, the firstlens 10 a is a black-and-white lens configured for capturing mono images(i.e. black-and-white images). The second lens 10 is a color lensconfigured for capturing color images.

The storage unit 15 may be one or a combination of a stationary ormobile random access memory (RAM), a read-only memory (ROM), a flashmemory, a hard drive or other similar devices. The storage unit 15 iscoupled to the first lens 10 a and the second lens 10 b for storingimages captured thereby.

The processing unit 20 may be, for example, a central processing unit(CPU) or other programmable devices for general purpose or specialpurpose such as a microprocessor and a digital signal processor (DSP), aprogrammable controller, an application specific integrated circuit(ASIC), a programmable logic device (PLD) or other similar devices or acombination of above-mentioned devices. The processing unit 20 iscoupled to the first lens 10 a, the second lens 10 b, and the storageunit 15, and includes, for example, an image capturing module 122, amode selecting module 124, and an image processing module 126 forperforming a hybrid image processing operation on images captured by theimage capturing device 100. Detail steps of the hybrid image processingmethod performed by the image capturing device 100 would be illustratedby the embodiments as follows.

FIG. 2 illustrates a flowchart of a hybrid image processing method of animage capturing device according to an embodiment of the disclosure, andthe method in FIG. 2 may be implemented by the components of the imagecapturing device 100 in FIG. 1.

Referring to both FIG. 1 and FIG. 2, the image capturing module 122 ofthe image capturing device 100 first captures a scene by using the firstlens 10 a and the second lens 10 b so as to respectively generate a monoimage and a color image of the scene (Step S202). In other words, when auser desires to capture an image of the scene by using the imagecapturing device 100, the image capturing module 122 would generate amono image corresponding to the first lens 10 a and a color imagecorresponding to the second lens 10 b.

Next, the mode selecting module 124 selects one of a mono image-basedmode, a color image-based mode, and a color image-only mode dynamicallyaccording to a content of the scene (Step S204), and the imageprocessing module 126 generates an output image according to a modeselected by the mode selecting module 124 (Step S206). Herein, the monoimage-based mode is to use the mono image as a primary image and thecolor image as a supplementary image, and the image processing module126 produces the output image by adding color image data of the colorimage onto the mono image. On the other hand, the color image-based modeis to use the color image as a primary image and the mono image as asupplementary image, and the image processing module 126 produces theoutput image by adding mono image data of the mono image onto the colorimage. The color image-only mode is to only use the color image toproduce the output image.

To be specific, the mono-based mode is suitable to preserve thebrightness of texture details in a planar scene, whereas the color-basedmode is suitable to preserve color schemes in a scene with mid-leveldepth variation. Hence, in the present embodiment, the mode selectingmodule 124 would adaptively select an image processing mode based onimage data captured by the image capturing module 122 so as to allow theimage processing module 126 to produce an image with decent quality. Forexample, FIG. 3 illustrates a flowchart of a hybrid image processingmethod of an image capturing device according to an embodiment of thedisclosure, and the method in FIG. 3 may be implemented by thecomponents of the image capturing device 100 in FIG. 1.

Referring to both FIG. 1 and FIG. 3, as in Step S202, the imagecapturing module 122 of the image capturing device 100 captures a sceneby using the first lens 10 a and the second lens 10 b so as torespectively generate a mono image and a color image of the scene (StepS302). Since the first image 10 a and the second image 10 b captureimages of the same scene from different viewing angles, the modeselecting module 124 could obtain depth range and depth variation of thescene based on a disparity between the two images to dynamically selecta mode to produce an output image in the follow-up steps.

First, the mode selecting module 124 determines whether the depth rangeis greater than a distance threshold or the depth variation is less thana first depth variation threshold (Step S304). If yes, the modeselecting module 124 would select the mono image-based mode to producethe output image (Step S306). To be specific, when the object distanceof a target object is far (e.g. greater than 3 m) or the depth variationis small (a planar scene with depth variation less than, for example, 10pixels), the difference between the images captured by the two lens isnot significant. In such scenario, the mode selecting module 124 wouldselect the mono image-based mode in which the mono image with moredetails is set as a primary image and the color image is set as asupplementary image. The image processing module 126 would produce theoutput image based on the processing approach set in such mode. In thepresent embodiment, the image processing module 126 may combine colorimage data of the color image into the mono image. The image processingmodule 126 may set pixels in the mono image (referred to as “lumapixels” hereinafter) as references, search for the corresponding pixelsfrom the color image (referred to as “related color pixels”), andcombine the pixel data of each correspondence. For example, the imageprocessing module 126 may combine related color data of the relatedcolor pixels into luma data of the luma pixels to produce a color imagewith rich details. It should be noted that, the aforesaid imagecombination approach is only for illustrative purposes. The disclosureis not limited to any particular image combination approach.

On the other hand, when the mode selecting module 124 determines thatthe depth range is not greater than the distance threshold and the depthvariation is not less than the first depth variation threshold in StepS304, the mode selecting module 124 further determines whether the depthvariation is less than a second depth variation threshold (Step S308).If so, the mode selecting module 124 would select the color image-basedmode to produce the output image (Step S310). Herein the second depthvariation is greater than the first depth variation and may be, forexample, 20 pixels. To be specific, when there exists certain level ofdepth variation in a scene, some regions appearing in the mono imagecould be occluded in the color image. In other words, no region andcolor data corresponding to certain regions in the mono image could befound from the color image. In such scenario, the mode selecting module124 would select the color image-based mode in which the color image isset as a primary image and the mono image is set as a supplementaryimage. The image processing module 126 would produce the output imagebased on the processing approach set in such mode. In the presentembodiment, the image processing module 126 may combine mono image dataof the mono image into the color image. The image processing module 126may set pixels in the color image (referred to as “color pixels”hereinafter) as references, search for the corresponding pixels from themono image (referred to as “related luma pixels”), and combine the pixeldata of each correspondence. For example, the image processing module126 may combine related luma data of the related luma pixels into colordata of the color pixels to produce a color image with a certain levelof details.

However, when the mode selecting module 124 determines that the depthvariation is not less than the second depth variation threshold in StepS308, the mode selecting module 124 would select the color image-onlymode to produce the output image (Step S312). That is, when the depthvariation exceeds a tolerable range, the probability of getting occludedregions could be higher and the area thereof could be larger. Hence,image combination may potentially cause discontinuities in such regions.In such scenario, the mode selecting module 124 would generate theoutput image by only using the color image data.

The proposed hybrid image processing method of an image capturing devicecould be illustrated as a functional flowchart according to anembodiment of the disclosure.

Referring to FIG. 1 and FIG. 4, in the proposed method, the imagecapturing device 122 of the image capturing device 100 captures a sceneby using the first lens 10 a and the second lens 10 b so as torespectively generate a mono image ImgBW and a color image ImgC. Next,the mode selecting module 124 performs mode switching MS according tothe mono image ImgBW and the color image ImgC so as to switch to one ofa mono image-based mode M1, a color image-based mode M2, and a colorimage-only mode M3 dynamically. The mono image-based mode M1 is to setthe mono image ImgBW as a primary image and the color image ImgC as asupplementary image and to add color image data of the color image ImgConto the mono image ImgBW. The color image-based mode M2 is to set thecolor image ImgC as a primary image and the mono image ImgBW as asupplementary image and to add mono image data of the mono image ImgBWonto the color image ImgC. The color-only mode M3 is to only process ordirectly output the color image ImgC. The image processing module 126would generate an output image Img according to the mode selected by themode selecting module 124.

In view of the foregoing, in the proposed image capturing device and thehybrid image processing method thereof, dual lenses are used forcapturing a mono image and a color image of a scene. An output image isgenerated by dynamically switching between the mono image-based mode,the color image-based mode, and the color image-only mode. Based on theadvantages of a mono image and a color image in different scenes, thedisclosure would adaptively switch to an image processing mode suitablefor the current scene, which not only provide output images with highquality but also accentuate the slimness of the image capturing device.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the structure of the presentinvention without departing from the scope or spirit of the invention.In view of the foregoing, it is intended that the present inventioncover modifications and variations of this invention provided they fallwithin the scope of the following claims and their equivalents.

What is claimed is:
 1. A hybrid image processing method of an imagecapturing device having a first lens and a second lens comprising:capturing a scene by using the first lens and the second lens so as torespectively generate a mono image and a color image of the scene; andselecting one of a mono image-based mode, a color image-based mode, anda color image-only mode dynamically according to a content of the sceneso as to generate an output image, wherein the mono image-based mode isto produce the output image by adding color image data of the colorimage onto the mono image, the color image-based mode is to produce theoutput image by adding mono image data of the mono image onto the colorimage, and the color image-only mode is to produce the output image byonly using the color image.
 2. The method according to claim 1, whereinthe step of selecting one of the mono image-based mode, the colorimage-based mode, and the color image-only mode dynamically according tothe content of the scene comprises: obtaining depth range and depthvariation of the scene according to the mono image and the color image;and selecting one of the mono image-based mode, the color image-basedmode, and the color image-only mode dynamically according to at leastone of the depth range and the depth variation.
 3. The method accordingto claim 2, wherein the step of selecting one of the mono image-basedmode, the color image-based mode, and the color image-only modedynamically according to at least one of the depth range and the depthvariation comprises: determining whether the depth range is greater thana distance threshold or the depth variation is less than a first depthvariation threshold; and when the depth range is greater than thedistance threshold or the depth variation is less than the first depthvariation threshold, selecting the mono image-based mode.
 4. The methodaccording to claim 3, wherein the step of selecting one of the monoimage-based mode, the color image-based mode, and the color image-onlymode dynamically according to at least one of the depth range and thedepth variation comprises: when the depth range is not greater than thedistance threshold and the depth variation is not less than the firstdepth variation threshold, determining whether the depth variation isless than a second depth variation threshold, wherein the second depthvariation threshold is greater than the first depth variation threshold;and when the depth variation is less than the second depth variationthreshold, selecting the color image-based mode.
 5. The method accordingto claim 4, wherein the step of selecting one of the mono image-basedmode, the color image-based mode, and the color image-only modedynamically according to at least one of the depth range and the depthvariation comprises: when the depth variation is not less than thesecond depth variation threshold, selecting the color image-only mode.6. The method according to claim 1, wherein the step of adding the colorimage data of the color image onto the mono image comprises: searchingfor a plurality of related color pixels from the color image, whereinthe related color pixels are associated with luma pixels in the monoimage; and combining related color data of the related color pixels intoluma data of the luma pixels.
 7. The method according to claim 1,wherein the step of adding the mono image data onto the color imagecomprises: searching for a plurality of related luma pixels from themono image, wherein the related luma pixels are associated with colorpixels in the color image; and combining related luma data of therelated luma pixels into color data of the color pixels.
 8. An imagecapturing device comprising: a first lens; a second lens; a storageunit, coupled to the first lens and the second lens and storing imagescaptured by the first lens and the second lens; and at least oneprocessing unit, coupled to the first lens, the second lens, and thestorage unit and comprising a plurality of modules, wherein the modulescomprise: an image capturing module, capturing a scene by using thefirst lens and the second lens so as to respectively generate a monoimage and a color image of the scene; a mode selecting module, selectingone of a mono image-based mode, a color image-based mode, and a colorimage-only mode dynamically according to a content of the scene; and animage processing module, generating an output image according to a modeselected by the mode selecting module, wherein the mono image-based modeis to produce the output image by adding color image data of the colorimage onto the mono image, the color image-based mode is to produce theoutput image by adding mono image data of the mono image onto the colorimage, and the color image-only mode is to produce the output image byonly using the color image.
 9. The image capturing device according toclaim 8, wherein the mode selecting module obtains depth range and depthvariation of the scene according to the mono image and the color imageand selects one of the mono image-based mode, the color image-basedmode, and the color image-only mode dynamically according to at leastone of the depth range and the depth variation.
 10. The image capturingdevice according to claim 8, wherein the mode selecting moduledetermines whether the depth range is greater than a distance thresholdor the depth variation is less than a first depth variation threshold,and if yes, the mode selecting module selects the mono image-based mode.11. The image capturing device according to claim 10, wherein when thedepth range is not greater than the distance threshold and the depthvariation is not less than the first depth variation threshold, the modeselecting module determines whether the depth variation is less than asecond depth variation threshold, wherein the second depth variationthreshold is greater than the first depth variation threshold, and ifyes, the mode selecting module selects the color image-based mode. 12.The image capturing device according to claim 11, wherein when the depthvariation is not less than the second depth variation threshold, themode selecting module selects the color image-only mode.
 13. The imagecapturing device according to claim 8, wherein when the mode selectingmodule selects the mono image-based mode, the image processing modulesearches for a plurality of related color pixels from the color imageand combines related color data of the related color pixels into lumadata of the luma pixels, wherein the related color pixels are associatedwith luma pixels in the mono image.
 14. The image capturing deviceaccording to claim 8, wherein when the mode selecting module selects thecolor image-based mode, the image processing module searches for aplurality of related luma pixels from the mono image and combinesrelated luma data of the related luma pixels into color data of thecolor pixels, wherein the related luma pixels are associated with colorpixels in the color image.